home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
FishMarket 1.0
/
FishMarket v1.0.iso
/
fishies
/
301-325
/
disk_319
/
cnewssrc
/
cnews.orig.lzh
/
man
/
newsbatch.man
< prev
next >
Wrap
Text File
|
1989-06-28
|
12KB
|
265 lines
NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888)))) CCCC NNNNeeeewwwwssss ((((22228888 MMMMaaaayyyy 1111999988889999)))) NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888))))
NNNNAAAAMMMMEEEE
sendbatches, batchsplit - news batching to other sites
batcher, batchih, batchsm - news-batch preparation
comp, nocomp, compcun, compc7, compb - news-batch
compression
c7encode, bencode - compressed-news-batch encoding
viauux, viainews, viamail, viapmail - news-batch
transmission
SSSSYYYYNNNNOOOOPPPPSSSSIIIISSSS
////uuuussssrrrr////lllliiiibbbb////nnnneeeewwwwssssbbbbiiiinnnn////bbbbaaaattttcccchhhh////sssseeeennnnddddbbbbaaaattttcccchhhheeeessss [ site ... ]
............////bbbbaaaattttcccchhhhsssspppplllliiiitttt batchsize
............////bbbbaaaattttcccchhhheeeerrrr [ ----dddd directory ] listfile
............////bbbbaaaattttcccchhhhiiiihhhh [ ----dddd directory ] listfile
............////bbbbaaaattttcccchhhhssssmmmm [ ----dddd directory ] listfile
............////ccccoooommmmpppp
............////nnnnooooccccoooommmmpppp
............////ccccoooommmmppppccccuuuunnnn
............////ccccoooommmmppppcccc7777
............////ccccoooommmmppppbbbb
............////cccc7777eeeennnnccccooooddddeeee
............////bbbbeeeennnnccccooooddddeeee [ file ]
............////vvvviiiiaaaauuuuuuuuxxxx site
............////vvvviiiiaaaaiiiinnnneeeewwwwssss site
............////vvvviiiiaaaammmmaaaaiiiillll site
............////vvvviiiiaaaappppmmmmaaaaiiiillll site
DDDDEEEESSSSCCCCRRRRIIIIPPPPTTTTIIIIOOOONNNN
_S_e_n_d_b_a_t_c_h_e_s administers batched transmission of news to
other sites. It should be run periodically, under userid
_n_e_w_s, by _c_r_o_n(8) or similar means. It prepares and sends
batches of news, subject to restrictions on available space
and length of outbound queues.
Each site that can have batches sent to it needs a _s_i_t_e
directory under /_u_s_r/_s_p_o_o_l/_n_e_w_s/_o_u_t._g_o_i_n_g. If _s_e_n_d_b_a_t_c_h_e_s
is invoked with specific _s_i_t_es given, it considers batching
for those sites, only, in that order. By default,
_s_e_n_d_b_a_t_c_h_e_s consults the _b_a_t_c_h_p_a_r_m_s file (see below) to
determine what to do: If there is a ////ddddeeeeffffaaaauuuulllltttt//// entry in
_b_a_t_c_h_p_a_r_m_s, _s_e_n_d_b_a_t_c_h_e_s will consider batching for all sites
that have directories in /_u_s_r/_s_p_o_o_l/_n_e_w_s/_o_u_t._g_o_i_n_g, in
oldest-first order by modification time of the directory.
If there is no ////ddddeeeeffffaaaauuuulllltttt//// entry, _s_e_n_d_b_a_t_c_h_e_s considers
batching for those sites named in _b_a_t_c_h_p_a_r_m_s, in the order
named.
To use the batcher, names of files to be sent to a specific
site should be appended to a _t_o_g_o file in its _s_i_t_e
directory. The batcher expects the lines in _t_o_g_o to have
two fields, a filename (as a full pathname) of an article
Page 1 (printed 6/29/89)
NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888)))) CCCC NNNNeeeewwwwssss ((((22228888 MMMMaaaayyyy 1111999988889999)))) NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888))))
and its size in bytes. A missing size field is arbitrarily
assumed to be a default average.
_S_e_n_d_b_a_t_c_h_e_s uses a number of auxiliary programs to do the
real work. The search path it uses to find them includes,
in order, the _s_i_t_e directory for the site in question,
/_u_s_r/_l_i_b/_n_e_w_s/_b_i_n, and /_u_s_r/_l_i_b/_n_e_w_s_b_i_n/_b_a_t_c_h. This permits
per-site and per-news-database overrides of the default
behaviors. _S_e_n_d_b_a_t_c_h_e_s provides all these programs with an
environment variable NNNNEEEEWWWWSSSSSSSSIIIITTTTEEEE, containing the name of the
site that batches are being prepared for, in case this is
useful in customization.
The names of most of the auxiliary programs, and some other
parameters, are taken from the file
/_u_s_r/_l_i_b/_n_e_w_s/_b_a_t_c_h_p_a_r_m_s, an ASCII text file. Empty lines
and lines starting with `#' are ignored. Other lines
specify the behavior for sites, one line per site. A site
line is six fields, separated by white space. The first
field is the site name; a line whose site name is ////ddddeeeeffffaaaauuuulllltttt////
specifies what parameters should be used for sites not
explicitly mentioned. (The presence or absence of such a
line also influences the behavior of _s_e_n_d_b_a_t_c_h_e_s when
invoked without arguments; see above.) The second field is
the size of batches to be prepared (before compression), in
bytes. The third field is the maximum number of batches to
be queued up for transmission to that site. The fourth,
fifth, and sixth fields are the names of the programs to be
used to build, compress, and transmit (respectively) batches
to that site.
For each site being considered for batches, _s_e_n_d_b_a_t_c_h_e_s
first determines whether there are in fact any articles to
be batched. Assuming there are, _s_e_n_d_b_a_t_c_h_e_s then finds the
_b_a_t_c_h_p_a_r_m_s line for that site and invokes _q_u_e_u_e_l_e_n (see
_n_e_w_s_a_u_x(8)) to find out how many batches are in the outbound
queue for it already. _S_e_n_d_b_a_t_c_h_e_s limits the number of
batches prepared to the minimum of the limits implied by
queue lengths and available space.
_S_e_n_d_b_a_t_c_h_e_s uses _b_a_t_c_h_s_p_l_i_t as necessary to slice chunks out
of the _t_o_g_o file, each chunk containing the _t_o_g_o lines for a
batch limited to the specified size. Exception: a single
article bigger than the specified size will still go out as
one batch.
Each chunk is then processed through a batch preparer
(typically _b_a_t_c_h_e_r), which assembles the articles into a
batch, a batch compressor (typically _c_o_m_p_c_u_n), which
performs compression and/or any other auxiliary processing
needed, and a batch transmitter (typically _v_i_a_u_u_x), which
sends the batch on its way (e.g. enqueues it for
Page 2 (printed 6/29/89)
NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888)))) CCCC NNNNeeeewwwwssss ((((22228888 MMMMaaaayyyy 1111999988889999)))) NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888))))
transmission).
Batch preparers in the standard distribution are:
batcher normal batching
batchih _i_h_a_v_e part of ihave/sendme
batchsm _s_e_n_d_m_e part of ihave/sendme
If _b_a_t_c_h_e_r is invoked with a ----dddd option, it _c_h_d_i_rs to the
_d_i_r_e_c_t_o_r_y first, and strips it off the beginning of any
article filename that starts with it. Since most articles
will come from /_u_s_r/_s_p_o_o_l/_n_e_w_s, specifying
----dddd ////uuuussssrrrr////ssssppppoooooooollll////nnnneeeewwwwssss to _b_a_t_c_h_e_r will speed it up noticeably.
Since _s_e_n_d_b_a_t_c_h_e_s does this, other preparers had better be
ready to ignore an initial ----dddd option.
Batch compressors in the standard distribution are:
comp ordinary compression
compcun compression plus the silly B-news-compatible
`#! cunbatch' header
nocomp pass data straight through, no compression
compc7 compression plus 7-bit encoding using _c_7_e_n_c_o_d_e
compb compression plus very conservative 6-bit
encoding using _b_e_n_c_o_d_e
Batch transmitters in the standard distribution are:
viauux normal transmission via UUCP
viainews feed the batch back to _i_n_e_w_s, ignoring the
_s_i_t_e argument (normally useful only for
ihave/sendme)
viamail mail the batch to _s_i_t_e!!!!rrrrnnnneeeewwwwssss
viapmail mail the batch to _s_i_t_e!!!!rrrrnnnneeeewwwwssss, attempting to
protect an unencoded batch against the
vagaries of mailers
_C_7_e_n_c_o_d_e encodes 8-bit data from standard input into a 7-bit
form optimized for transmission by uucp `f' protocol. The
encoding is complex and bizarre.
_B_e_n_c_o_d_e takes 8-bit data from the named source file (the
default is standard input) and encodes it using only the
Page 3 (printed 6/29/89)
NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888)))) CCCC NNNNeeeewwwwssss ((((22228888 MMMMaaaayyyy 1111999988889999)))) NNNNEEEEWWWWSSSSBBBBAAAATTTTCCCCHHHH((((8888))))
ASCII characters ``A'' - ``Z'', ``a'' - ``z'', ``0'' -
``9'', ``+'', and ``-''. The ASCII characters blank,
newline, and ``/'' also appear in the encoded file, but do
not represent encoded bits. The encoded file is terminated
with a byte count and cyclic redundancy check for detecting
corrupted files. This ought to suffice to get data through
almost any network.
_S_e_n_d_b_a_t_c_h_e_s logs some information about sites with backlogs
in /_u_s_r/_l_i_b/_n_e_w_s/_b_a_t_c_h_l_o_g, keeping the previous _b_a_t_c_h_l_o_g in
_b_a_t_c_h_l_o_g._o and the one before that in _b_a_t_c_h_l_o_g._o_o. This is
intended to help detection and diagnosis of flow problems.
FFFFIIIILLLLEEEESSSS
/usr/lib/news/LOCKbatch lock for _s_e_n_d_b_a_t_c_h_e_s
/usr/lib/news/LOCK overall news lock (used by _b_a_t_c_h_s_p_l_i_t)
/usr/lib/news/L.* lock temporaries
/usr/spool/news/out.going/* batch directories
/usr/lib/news/batchparms parameter file
also see text
SSSSEEEEEEEE AAAALLLLSSSSOOOO
inews(1), compress(1), uux(1), relaynews(8), rnews(8)
HHHHIIIISSSSTTTTOOOORRRRYYYY
Written at University of Toronto as part of the C News
project. A number of the ideas came from Chris Lewis, who
was then with Motorola. _B_e_n_c_o_d_e written at University of
Waterloo by Reg Quinton and Ken Lalonde.
BBBBUUUUGGGGSSSS
_B_a_t_c_h_s_p_l_i_t does not count the `#! rnews _n_n_n_n' headers inside
batches when computing batch lengths.
Ihave/sendme processing is a bit of a kludge; in particular,
applying _b_a_t_c_h_s_p_l_i_t to a file full of article IDs just
happens to work okay.
_S_e_n_d_b_a_t_c_h_e_s estimates available space without considering
the effects of compression; this is usually too
conservative.
_V_i_a_p_m_a_i_l is obsolescent; _v_i_a_m_a_i_l with a batch encoded with
_b_e_n_c_o_d_e is vastly superior.
Page 4 (printed 6/29/89)